Null Results Research Articles

Replication of null results: Absence of evidence or evidence of absence?

In several large-scale replication projects, statistically non-significant results in both the original and the replication study have been interpreted as a 'replication success.' Here, we discuss the logical problems with this approach: Non-significance in both studies does not ensure that the studies provide evidence for the absence of an effect and 'replication success' can virtually always be achieved if the sample sizes are small enough. In addition, the relevant error rates are not controlled. We show how methods, such as equivalence testing and Bayes factors, can be used to adequately quantify the evidence for the absence of an effect and how they can be applied in the replication setting. Using data from the Reproducibility Project: Cancer Biology, the Experimental Philosophy Replicability Project, and the Reproducibility Project: Psychology we illustrate that many original and replication studies with 'null results' are in fact inconclusive. We conclude that it is important to also replicate studies with statistically non-significant results, but that they should be designed, analyzed, and interpreted appropriately.

The Critical Role of Implementing Partners: Evidence From Training Micro and Small Enterprises Across 12 Financial Service Providers

Despite access to financial resources, many micro and small-scale entrepreneurs struggle to grow their businesses due to management inefficiencies. This study analyses a globally recognized ILO business training initiative in Indonesia, involving 12 financial service providers in 2018. Using an RCT and panel data from 3,975 clients to study the impacts of the business training we find largely null results across a large range of outcomes. Exploiting variation across financial institutions implementing the program, we show that one financial service provider demonstrated significant improvements in its clients’ behaviours. This uneven outcome suggests that the efficacy of large-scale interventions critically depends on the quality of implementation by individual partners. The overarching conclusions advise restraint in broadly expanding these programs, while simultaneously highlighting the crucial role of partner selection in the successful implementation of these initiatives.

A challenge to the expected: Lack of longitudinal associations between the early caregiving environment, executive functions in toddlerhood, and self-regulation at 6 years.

Previous research and theory indicate an importance of the quality of the early caregiving environment in the development of self-regulation. However, it is unclear how attachment security and maternal sensitivity, two related but distinct aspects of the early caregiving environment, may differentially predict self-regulation at school start and whether a distinction between hot and cool executive function is informative in characterizing such predictions through mediation. In a 5-year longitudinal study (n=108), we examined these associations using measures of maternal sensitivity and attachment security at 10-12 months, executive function at 4 years, and self-regulation at 6 years. Surprisingly, and despite methodological rigor, we found few significant bivariate associations between the study variables. We found no credible evidence of a longitudinal association between maternal sensitivity or attachment security in infancy and self-regulation at 6 years, or between executive function at 4 years and self-regulation at 6 years. The lack of bivariate longitudinal associations precluded us from building mediation models as intended. We discuss our null findings in terms of their potential theoretical implications, as well as how measurement type, reliability, and validity, may play a key role in determining longitudinal associations between early caregiving factors and later self-regulation and related abilities. RESEARCH HIGHLIGHTS: The early caregiving environment has been implicated in the development of later self-regulation, which includes more basic skills, such as hot and cool executive functions (EF). In a 5-year longitudinal study, with a sample of 108 children, we rigorously measured aspects of early caregiving, EF, and self-regulation. We found no significant longitudinal associations between early caregiving and self-regulation at 6 years, nor between EF at 4 years and self-regulation at 6 years. These null results highlight the complexity of modeling self-regulation development and raise critical questions about general methodological conventions within self-regulation development research.

Aspirin Metabolites and Mammographic Breast Density in Premenopausal Women.

Studies investigating the associations of self-reported aspirin use and mammographic breast density (MBD) have reported conflicting results. We, therefore, investigated the associations of aspirin metabolites, with MBD in premenopausal women. We performed this study on 705 premenopausal women who had fasting blood draw for metabolomic profiling. We performed covariate-adjusted linear regression models to calculate the least squares means of volumetric measures of MBD (volumetric percent density (VPD), dense volume (DV), and non-dense volume (NDV)) by quartiles of aspirin metabolites (salicyluric glucuronide, 2-hydroxyhippurate (salicylurate), salicylate, and 2,6-dihydroxybenzoic acid). Approximately 13% of participants reported taking aspirin in the past 12 months. Aspirin users had higher levels of 2-hydroxyhippurate (salicylurate), salicylate, and salicyluric glucuronide (peak area) than non-users, but only mean peak area of salicyluric glucuronide increased by both dose (1-2 tabs per day=1,140,663.7, and ≥3 tabs per day=1,380,476.0) and frequency (days per week: 1 day=888,129.3, 2-3 days=1,199,897.9 and ≥4 days=1,654,637.0). Aspirin metabolites were not monotonically associated with VPD, DV, or NDV. Given the null results, additional research investigating the associations of aspirin metabolites in breast tissue and MBD is necessary. Elucidating the determinants of MBD, a strong risk factor for breast cancer, can play an important role in breast cancer prevention. Future studies should determine the associations of non-aspirin non-steroidal anti-inflammatory drug metabolites with MBD.

Exploring human-artificial intelligence interactions in a negative pragmatic trial of computer-aided polyp detection

Background and aimsThe progress of artificial intelligence (AI) in endoscopy is at a crossroads. The positive results of randomized controlled trials (RCT) of computer-aided detection (CADe) have not been replicated in multiple pragmatic CADe trials, including ours. This gap between efficacy and effectiveness remains to be understood. We surveyed and interviewed our trial’s colonoscopists to gain insight into human-AI interactions. MethodsWe used a sequential, mixed-methodology design. Following the trial, we administered Survey 1 focusing on attitudes and beliefs before and after trying CADe. The trial’s null results were disclosed, and we then administered Survey 2 and conducted open-ended interviews, focusing on reactions to the null results. Responses were analyzed overall and by baseline adenoma detection rate (ADR) tertile. We identified key themes utilizing thematic analysis and qualitative software. ResultsNearly all colonoscopists responded (22 and 21 of 24 [92%, 88%] for Surveys 1 and 2). Most (96%) regarded endoscopic ability as critical to their professional identity. Large majorities conveyed trust in and enthusiasm for AI before and after trying CADe (82-87%), and desired to have CADe available (72%). Nearly two-thirds (62%) were surprised by the null results. There were few differences by ADR. No unifying explanation for the null results emerged from surveys or individual interviews. Colonoscopists expressed a range of expectations for AI in endoscopy. ConclusionsLack of enthusiasm or mistrust of AI/CADe do not explain our pragmatic CADe trial’s null results. AI may need to target dimensions beyond optical recognition to realize its promise in endoscopy.

Cash transfers and mental health in Egypt

The relationship between poverty and mental health is complex. Conditional cash transfers are seen as an important policy tool in reducing poverty and fostering social protection. Evidence on the impact on mental health is mixed. In this study, we assess the causal impact of the Egypt’s conditional cash transfer (CCT) programme Takaful on the main beneficiaries’ mental health. Using a regression discontinuity design, we find that receiving the Takaful CCT does not have a significant impact on the anxiety levels of mothers in our sample. In addition, we do not find much supporting evidence that the programme has an heterogeneous impact on anxiety levels. We discuss possible explanations behind these null results.

Optogenetic behavioral studies in depression research: A systematic review

Optogenetics has made substantial contributions to our understanding of the mechanistic underpinnings of depression. This systematic review employs quantitative analysis to investigate the impact of optogenetic stimulation in mice and rats on behavioral alterations in social interaction, sucrose consumption, and mobility. The review analyses optogenetic behavioral studies using standardized behavioral tests to detect behavioral changes induced via optogenetic stimulation in stressed or stress-naive mice and rats. Behavioral changes were evaluated as either positive, negative, or not effective. The analysis comprises the outcomes of 248 behavioral tests of 168 studies described in 37 articles, including negative and null results. Test outcomes were compared for each behavior, depending on the animal cohort, applied type of stimulation and the stimulated neuronal circuit and cell type. The presented synthesis contributes toward a comprehensive picture of optogenetic behavioral research in the context of depression.

The effects of adding social-emotional learning to a comprehensive education intervention in El Salvador on teacher well-being: a mixed methods evaluation

ABSTRACT This article employs a mixed-method approach to assess the effects of El Salvadorian Integrated Systems of Full-Time Inclusive Schools (SI-EITP), which offers in-service teacher professional development (TPD) combined with a socioemotional learning intervention, on teacher well-being. Findings from the cluster-randomized controlled trial with no baseline show null results for most teacher well-being outcomes analyzed. Interviews suggest limited effects on teachers’ mindfulness and emotion regulation and indicate that teachers’ participation in TPD was potentially increasing their stress levels and emotion exhaustion. We recommend that TPD program designs consider how the delivery mode and intensity may influence teachers’ stress.

Search for the lepton flavor violating τ → 3μ decay in proton-proton collisions at [formula omitted]

A search for the lepton flavor violating τ→3μ decay is performed using proton-proton collision events at a center-of-mass energy of 13 TeV collected by the CMS experiment at the LHC in 2017–2018, corresponding to an integrated luminosity of 97.7 fb−1. Tau leptons produced in both heavy-flavor hadron and W boson decays are exploited in the analysis. No evidence for the decay is observed. The results of this search are combined with an earlier null result based on data collected in 2016 to obtain a total integrated luminosity of 131 fb−1. The observed (expected) upper limits on the branching fraction B(τ→3μ) at confidence levels of 90 and 95% are 2.9×10−8 (2.4×10−8) and 3.6×10−8 (3.0×10−8), respectively.

Is artificial intelligence the great filter that makes advanced technical civilisations rare in the universe?

This study examines the hypothesis that the rapid development of Artificial Intelligence (AI), culminating in the emergence of Artificial Superintelligence (ASI), could act as a "Great Filter" that is responsible for the scarcity of advanced technological civilisations in the universe. It is proposed that such a filter emerges before these civilisations can develop a stable, multiplanetary existence, suggesting the typical longevity (L) of a technical civilization is less than 200 years. Such estimates for L, when applied to optimistic versions of the Drake equation, are consistent with the null results obtained by recent SETI surveys, and other efforts to detect various technosignatures across the electromagnetic spectrum. Through the lens of SETI, we reflect on humanity's current technological trajectory – the modest projections for L suggested here, underscore the critical need to quickly establish regulatory frameworks for AI development on Earth and the advancement of a multiplanetary society to mitigate against such existential threats. The persistence of intelligent and conscious life in the universe could hinge on the timely and effective implementation of such international regulatory measures and technological endeavours.

Sustaining attention in visuomotor timing is associated with location-based binding

Maintaining focus of attention over prolonged periods can be challenging, especially when the target stimulus is absent from the temporal sequence. Prior research has shown that a temporal attentional cue filling in the temporal blank can improve sustained attention: in a sustained visual attention task requiring synchronizing finger tapping with a temporally regular sequence composed of brief flash disks interleaved with blank periods, task performance was improved when a continuous fixation point that served as a temporal attentional cue was presented superimposed on the disk stimulus. To test the hypothesis that binding the temporal attentional cue with the target temporal sequence by spatial overlapping is crucial for enhancing sustained attention, the present study conducted a series of three experiments that deconstructed the bound connection between the cue and the sequence stimulus. In Experiment 1, the cue was placed above or below a flash disk. In Experiment 2, the cue was between two vertically arranged flash disks. In Experiment 3, the cue was in a flash ring. No significant effect of sustained attention improvement was found in any of the three experiments. Experiment 4 further replicated these null results and the previously observed effect of sustained attention improvement when the temporal cue was superimposed on the sequence stimulus. Our finding demonstrates that binding by spatial overlapping during the temporal blank when the sequence stimulus is absent is critical for enhancing sustained attention, which should be beneficial for improving performance across a broader range of tasks that require prolonged maintenance of attention.

From apprentice to president? Entertainment TV and US elections

This article studies the importance of entertainment TV for the selection of political leaders in the context of an important case study: Donald Trump's win in the 2016 presidential election and his previous role as host of the popular TV show “The Apprentice.” We find a positive correlation between TV ratings of The Apprentice and the county-level Republican vote share in 2016, but this correlation vanishes once we control for pre-existing voting and NBC viewership patterns. This null result is robust to different model specifications, measures of exposure to The Apprentice, and an extensive investigation of heterogeneous effects. Viewership of The Apprentice is also unrelated to Congressional election results, as well as support for Trump in survey data and the Republican primaries. These findings highlight the context-dependent importance of television for political leadership.

Effects of neural oscillation power and phase on discrimination performance in a visual tilt illusion

Neural oscillations reflect fluctuations in the relative excitation/inhibition of neural systems1,2,3,4,5 and are theorized to play a critical role in canonical neural computations6,7,8,9 and cognitive processes.10,11,12,13,14 These theories have been supported by findings that detection of visual stimuli fluctuates with the phase of oscillations prior to stimulus onset.15,16,17,18,19,20,21,22,23 However, null results have emerged in studies seeking to demonstrate these effects in visual discrimination tasks,24,25,26,27 raising questions about the generalizability of these phenomena to wider neural processes. Recently, we suggested that methodological limitations may mask effects of phase in higher-level sensory processing.28 To test the generality of phasic influences on perception requires a task that involves stimulus discrimination while also depending on early sensory processing. Here, we examined the influence of oscillation phase on the visual tilt illusion, in which a center grating has its perceived orientation biased away from the orientation of a surround grating29 due to lateral inhibitory interactions in early visual processing.30,31,32 We presented center gratings at participants' subjective vertical angle and had participants report whether the grating appeared tilted clockwise or counterclockwise from vertical on each trial while measuring their brain activity with electroencephalography (EEG). In addition to effects of alpha power and aperiodic slope, we observed robust associations between orientation perception and alpha and theta phase, consistent with fluctuating illusion magnitude across the oscillatory cycle. These results confirm that oscillation phase affects the complex processing involved in stimulus discrimination, consistent with its purported role in canonical computations that underpin cognition.

Alpha-Band Frequency and Temporal Windows in Perception: A Review and Living Meta-analysis of 27 Experiments (and Counting).

Temporal windows in perception refer to windows of time within which distinct stimuli interact to influence perception. A simple example is two temporally proximal stimuli fusing into a single percept. It has long been hypothesized that the human alpha rhythm (an 8- to 13-Hz neural oscillation maximal over posterior cortex) is linked to temporal windows, with higher frequencies corresponding to shorter windows and finer-grained temporal resolution. This hypothesis has garnered support from studies demonstrating a correlation between individual differences in alpha-band frequency (IAF) and behavioral measures of temporal processing. However, nonsignificant effects have also been reported. Here, we review and meta-analyze 27 experiments correlating IAF with measures of visual and audiovisual temporal processing. Our results estimate the true correlation in the population to be between .39 and .53, a medium-to-large effect. The effect held when considering visual or audiovisual experiments separately, when examining different IAF estimation protocols (i.e., eyes open and eyes closed), and when using analysis choices that favor a null result. Our review shows that (1) effects have been internally and independently replicated, (2) several positive effects are based on larger sample sizes than the null effects, and (3) many reported null effects are actually in the direction predicted by the hypothesis. A free interactive web app was developed to allow users to replicate our meta-analysis and change or update the study selection at will, making this a "living" meta-analysis (randfxmeta.streamlit.app). We discuss possible factors underlying null reports, design recommendations, and open questions for future research.

Parietal theta burst TMS does not modulate bistable perception.

The role of the parietal cortex in perceptual awareness and in resolving perceptual ambiguity is unsettled. Early influential transcranial magnetic stimulation studies have revealed differences in conscious perception following parietal stimulation, fuelling the notion that parietal cortex causally contributes to resolving perceptual ambiguity. However, central to this conclusion is the reliability of the method employed. Several prior studies have revealed opposing effects, such as shortening, lengthening, or no effect on multistable perceptual transitions following parietal stimulation. Here we addressed the reliability of continuous theta-burst stimulation (cTBS) on parietal cortex on the perception of bistable stimuli. We conducted three cTBS experiments that were matched to prior experiments in terms of stimuli, stimulation protocol, and target site, and used a higher number of participants. None of our cTBS experiments replicated prior cTBS results. The only experiment using individual functional localizers led to weak effects, while the two others led to null results. Individual variability of motor cortex cTBS did not predict parietal cTBS effects. In view of recent reports of highly variable cTBS effects over motor cortex, our results suggest that cTBS is particularly unreliable in modulating bistable perception when applied over parietal cortex.

Lyα at Cosmic Dawn with a Simulated Roman Grism Deep Field

The slitless grism on the Nancy Grace Roman Space Telescope will enable deep near-infrared spectroscopy over a wide field of view. We demonstrate Roman's capability to detect Lyα galaxies at z > 7 using a multiple position angle (PA) observational strategy. We simulate Roman grism data using a realistic foreground scene from the COSMOS field. We also input fake Lyα galaxies spanning redshift z = 7.5–10.5 and a line-flux range of interest. We show how a novel data-cube search technique—CUBGRISM—originally developed for the Galaxy Evolution Explorer can be applied to Roman grism data to produce a Lyα flux-limited sample without the need for continuum detections. We investigate the impact of altering the number of independent PAs and exposure time. A deep Roman grism survey with 25 PAs and a total exposure time of 70 hr can achieve Lyα line depths comparable to the deepest z = 7 narrowband surveys (L Lyα ≳ 1043 erg s−1). Assuming a null result, where the opacity of the intergalactic medium (IGM) remains unchanged from z ∼ 7, this level of sensitivity will detect ∼400 deg−2 Lyα emitters from z = 7.25 to 8.75. A decline from this expected number density is the signature of an increasing neutral hydrogen fraction and the onset of reionization. Our simulations indicate that a deep Roman grism survey has the ability to measure the timing and magnitude of this decline, allowing us to infer the ionization state of the IGM and helping us to distinguish between models of reionization.

More pain, more gain? Extricating the effect of student involvement in rubric co-construction

This quasi-experiment tested whether co-constructing a writing rubric gives students an advantage over only using the co-constructed rubric. We used students’ perception and use of the rubric, and their writing performance as outcome variables. One hundred and twenty-six second-year university students who enrolled in an elective course participated. The experimental group (N = 63) co-constructed the rubric with the lecturer. The rubric was then distributed to both groups to facilitate their writing. A peer-feedback session was held for the students to use the rubric to offer and interpret feedback. Independent-samples t-tests revealed no significant differences between the two groups in terms of their perception of the rubric, use of the rubric and writing performance. The overall null result suggests that as a one-off activity, rubric co-construction does not bring extra benefits to the students involved.

Searching for Intermediate-mass Black Holes in Globular Clusters through Tidal Disruption Events

Intermediate-mass black holes (IMBHs) may be the link between stellar mass holes and the supermassive variety in the nuclei of galaxies, and globular clusters (GCs) may be one of the most promising environments for their formation. Here, we carry out a pilot study of the observability of tidal disruption events (TDEs) from 103 M ⊙ < M • < 105 M ⊙ IMBHs embedded in stellar cusps at the center of GCs. We model the long super-Eddington accretion phase and ensuing optical flare, and derive the disruption rate of main-sequence stars as a function of black hole mass and GC properties with the help of a 1D Fokker–Planck approach. The photospheric emission of the adiabatically expanding outflow dominates the observable radiation and peaks in the near-ultraviolet/optical bands, outshining the brightness of the (old) stellar population of GCs in Virgo for a period of months to years. A search for TDE events in a sample of nearly 4000 GCs observed at multiple epochs by the Next Generation Virgo Cluster Survey yields null results. Given our model predictions, this sample is too small to set stringent constraints on the present-day occupation fraction of GCs hosting IMBHs. Naturally, better simulations of the properties of the cluster central stellar distribution, TDE light curves, and rates, together with larger surveys of GCs are all needed to gain deeper insights into the presence of IMBHs in GCs.

The effect of negative valence on false memory formation in the Deese-Roediger-McDermott paradigm: A preregistered meta-analysis and preregistered replication.

Participants in the Deese-Roediger-McDermott (DRM) paradigm learn lists of words (e.g., bed, tired) associated with a nonpresented lure (i.e., sleep). In subsequent memory tests, individuals tend to report the nonlearned lures, that is, exhibiting false memories. Priorly, the DRM task has been criticized for not capturing the aversive nature of (clinically and forensically relevant) real-life memories. To obtain a robust estimate of the influence of negative versus neutral word lists on the DRM effect, we conducted both a preregistered meta-analysis (krecall = 49, nrecall = 2,209, krecognition = 75, nrecognition = 3,008, kresponsebias = 31, nresponsebias = 1,128) and replication (nfinal = 278) predicting increased false memories for negative valence in recall and recognition. For recall, we found significant frequentist evidence in the meta-analysis for a reversed valence effect (d = -0.18, i.e., reduced false memories for negative content vs. neutral), whereas the replication displayed null results (d = 0.03). For recognition, both the meta-analysis (d = 0.23) and replication (d = 0.35) showed that negative valence (vs. neutral) increased false memories. However, this effect may be confounded by shifts in response tendencies as controlling for response bias nullified the valence effect in our meta-analysis (dmeta = 0.05), and we found evidence for differential response bias in our replication (dreplica = 0.39). Hence, the effect of valence on false memory reports in the DRM may not represent a systematic difference in emotional information but instead depend on how memory is tested, and be partly attributable to differential response tendencies. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

"Catastrophic" set size limits on infants' capacity to represent objects: A systematic review and Bayesian meta-analysis.

Decades of research has revealed that humans can concurrently represent small quantities of three-dimensional objects as those objects move through space or into occlusion. For infants (but not older children or adults), this ability apparently comes with a significant limitation: when the number of occluded objects exceeds three, infants experience what has been characterized as a "catastrophic" set size limit, failing to represent even the approximate quantity of the hidden array. Infants' apparent catastrophic representational failures suggest a significant information processing limitation in the first years of life, and the evidence has been used as support for prominent theories of the development of object and numerical cognition. However, the evidence for catastrophic failure consists of individual small-n experiments that use null hypothesis significance testing to obtain null results (i.e., p>0.05). Whether catastrophic representational failures are robust or reliable across studies, methods, and labs is not known. Here we report a systematic review and Bayesian meta-analysis to examine the strength of the evidence in favor of catastrophic representational failures in infancy. Our analysis of 22 experiments across 12 reports, with a combined total of n=367 infants aged 10-20 months, revealed strong support for the evidence for catastrophic set size limits. A complementary analysis found moderate support for infants' success when representing fewer than four objects. We discuss the implications of our findings for theories of object and numerical cognitive development. RESEARCH HIGHLIGHTS: Previous work has suggested that infants are unable to concurrently represent four or more objects-a "catastrophic" set size limit. We reviewed this work and conducted a Bayesian meta-analysis to examine the robustness of this limit across individual small-n experiments. We found strong support for the evidence for catastrophic set size limits, and moderate support for infants' success when representing fewer than four objects.